
DRAWING PROCESSING APPARATUS , DRAWING PROCESSING METHOD, 
DRAWING PROCESSING PROGRAM AND TELECONFERENCE SYSTEM 

EQUIPPED THEREWITH 

Background of the Invention 

5 Field of the Invention 
[0001] 

The present invention relates to a drawing processing 
apparatus, a drawing processing method, a drawing 
processing program and a teleconference system equipped 
10 therewith. 

Description of the Related Art 
[0002] 

In recent years, process of a moving image by using a 
15 general-purpose PC has become general as broadband 
environment spreads and performance of a PC (personal 
computer) improves. For example, in a teleconference 
system etc., the following method is known as a drawing 
processing method in which a symbol, an arrow, a character, 
20 any writing, etc. on the display screen are added by a 
drawing input device etc. to superimpose upon a moving 
image when the moving image is displayed on a display 
screen outputted from a general-purpose PC. 
[0003] 

25 In a general-purpose PC in which an OS (operating 
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system) such as Windows (registered trademark) is used, a 
drawing processing method for superimposing drawn static 
images upon a moving image on a display screen to display a 
superimposed image is used. A drawing processing method in 
5 which a graphics accelerator is used is explained below. 
[0004] 

Fig. 1 is a diagram showing a conventional drawing 
processing apparatus 200 in which a graphics accelerator is 
used on an OS of a general-purpose PC. As shown in Fig. 1, 

10 a moving image which a user selects from among moving image 
data stored in a mass recording medium 211 is sent to a 
moving image reproducing section 212. Then, drawing 
processing is performed by a hardware drawing section 213 
of the graphics accelerator to display the selected moving 

15 image on a screen display device 216. On the other hand, 
input drawing data which a user inputs by using a drawing 
input device 214 is processed in an image drawing section 
215 to display on the screen display device 216 
concurrently with the moving image. 

20 [0005] 

As a result, as shown in Fig. 2, it can be seen that 
a moving image A (moving image of a flying bird) obtained 
by performing the hardware drawing in the graphics 
accelerator and an input drawing image (arrow) B processed 
25 by the image drawing section 215 on the OS independently of 
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this moving image are overlapped on a display screen of the 

screen display device 216. 

[0006] 

As a method in which a user is interactively 
5 associated when a moving image is displayed on a display 
screen, for example, a technique for combining moving image 
data with static image data previously stored according to 
input of the user, described in JP-A-7-306953, is known. 
[0007] 

10 JP-A-7-306953 is referred as a related art. 

[0008] 

In the conventional drawing processing apparatus 
described above, it is impossible to capture a moving image 
displayed on the screen display device 216 into a clipboard 
15 (memory) . This is because a screen capture which is a 
portion of functions of the OS cannot be performed since 
the OS does not directly control the hardware drawing 
section 213 as the graphics accelerator. 
[0009] 

20 Therefore, when a screen capture function of the OS 

is utilized, as shown in Fig. 3, the screen capture is 
performed in the image drawing section 215 directly managed 
by the OS, and only input drawing data is captured in the 
clipboard. Thus, on a screen captured in the clipboard, 

25 only the input drawing image (arrow) B is displayed on a 
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black screen in which moving image data is not present, as 

shown in Fig. 4. 

[0010] 

On the other hand, in a conventional drawing 
5 processing apparatus 200A of the case that setting of an OS 
is changed and drawing (writing) is performed on a moving 
image without using a graphics accelerator, as shown in 
Fig. 5, an image drawing section 215A which is directly 
controlled by the OS manages input drawing data and moving 
10 image data. As a result, screen capture of a moving image 
can be performed. 
[0011] 

However, in this case, as shown in Fig. 6, since 
moving image data (al, a2, a3, ...) are successively sent to 

15 the image drawing section 215A, input drawing data sent to 
the image drawing section 215A is overwritten by the moving 
image data (because exclusive control by the OS is not 
performed) and the input drawing data is overwritten and 
deleted. 

20 [0012] 

As explained above, in the conventional drawing 
processing apparatus, a function in which when a moving 
image is displayed on a display screen, a user writes a 
symbol, an arrow, a character, any writing, etc. on the 
25 display screen by a drawing input device etc. to combine 
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the symbol etc. with the moving image and displays in real 
time could not be balanced with a function in which the 
combined image is captured to be retained in a storage 
medium etc. and is displayed later. 
5 [0013] 

In a teleconference system etc., while a user writes 
a symbol, an arrow, a character, any writing, etc. by a 
drawing input device etc. rather than static image data 
stored previously, the user combines this input drawing 

10 data with a moving image. Thus, even in the case of 
applying a conventional combining technique (for example, 
JP-A-7-306953) for combining a static image with a moving 
image, processing capability is limited in a general- 
purpose PC, so that frame omission etc. occur. Therefore, 

15 it is difficult to perform smooth display of a moving 
image . 
[0014] 

Although it is considered that input drawing data is 
overwritten and drawn repeatedly on a moving image, the 

20 input drawing data is not continuous due to disappearance 
or output and an image becomes difficult to see due to 
flicker. Also, since it is necessary to repeat drawing of 
the input drawing data at high speed according to a change 
in moving images, processing of the PC becomes an overload. 

25 [0015] 
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Otherwise, it is considered that a window is created 
by a shape of a path of a pen with which an input drawing 
image is drawn and writing on its moving image is 
performed- Thus, although a window object can be displayed 
5 on the moving image by a management function of an OS, 
processing for forming the window by the pen path becomes 
an overload. In addition, the input drawing image breaks 
when the window forming processing does not catch up with 
the drawing. 

10 Summary of the Invention 

[0016] 

As objects that the invention is to solve, a problem 
that a function in which a user writes a symbol, an arrow, 
a character, any writing, etc. on the display screen 

15 generated in the conventional art described above by a 
drawing input device etc. to combine the symbol etc. with a 
moving image and displays in real time cannot be balanced 
with a function in which the combined image is captured to 
be retained in a storage medium etc. and is displayed later 

20 is given as one example. 
[0017] 

The invention provides a drawing processing apparatus 
having an image control section for cutting out a moving 
image as static image information in the moving image every 
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predetermined time and extracting input drawing static 
information from a drawn input image every the 
predetermined time, an image information storage section 
for storing the static image information in the moving 
5 image cut out by the image control section and the input 
drawing static information extracted by the image control 
section, an image combining section for combining the 
static image information in the moving image and the input 
drawing static information stored in the image information 
10 storage section to create combined image information, and 
an image drawing section for continuously outputting the 
combined image information. 
[0018] 

The invention also provides a drawing processing 
15 method having the steps of cutting out an image as static 
image information in a moving image from the moving image, 
extracting input drawing static information from a drawn 
input image, combining the static image information in the 
moving image obtained by the cutout and the input drawing 
20 static information obtained by the extraction to create 
combined image information, and outputting the combined 
image information, wherein the cutout of the static image 
information in the moving image and the extraction of the 
input drawing static information are repeated every 
25 predetermined time. 
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[0019] 

The invention provides a drawing processing program, 
causing a computer to perform an image control function of 
cutting out an image as static image information in a 
moving image from the moving image every predetermined time 
and extracting input drawing static information from a 
drawn input image every the predetermined time, and an 
image combining function of combining the static image 
information in the moving image cut out by the image 
control function and the input drawing static information 
extracted by the image control function to create combined 
image information . 
[0020] 

The invention provides a teleconference system, in 
which a plurality of participant terminals which 
participate in a conference are connected through a 
communication line, wherein a drawing processing apparatus 
having an image control section for cutting out an image as 
static image information in a moving image from the moving 
image every predetermined time and extracting input drawing 
static information from a drawn input image every the 
predetermined time, an image information storage section 
for storing the static image information in the moving 
image cut out by the image control section and the input 
drawing static information extracted by the image control 
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section, an image combining section for combining the 
static image information in the moving image and the input 
drawing static information stored in the image information 
storage section to create combined image information, and 
5 an image drawing section for continuously outputting the 
combined image information, is used as the participant 
terminal . 

Brief Description of the Drawings 

Fig. 1 is a diagram showing a configuration of a 
10 conventional drawing processing apparatus; 

Fig. 2 is a diagram showing a drawing image by the 
conventional drawing processing apparatus using an 
accelerator; 

Fig. 3 is a diagram showing another configuration 
15 example of a conventional drawing processing apparatus; 

Fig. 4 is a diagram showing a result of image capture 
by the drawing processing apparatus shown in Fig. 3; 

Fig. 5 is a diagram showing another configuration 
example of a conventional drawing processing apparatus; 
2 0 Fig. 6 is a diagram showing a drawing image by the 

conventional drawing processing apparatus without using an 
accelerator; 

Fig. 7 is a diagram showing a configuration of a 
drawing processing apparatus in an embodiment of the 
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invention; 

Fig. 8 is a diagram showing a drawing image by 
drawing processing to a moving image according to the 
invention; 

5 Fig. 9 is a flowchart showing a drawing processing 

procedure according to the invention; 

Fig. 10 is a diagram describing one example of a 
configuration of drawing data; 

Fig. 11 is a flowchart showing one example of a 
10 combining method of input drawing static data and moving 
image inside static image data; 

Figs. 12A to 12C are examples of a combined image by 
the combining method of Fig. 11; 

Fig. 13 is a flowchart showing details of a cutout 
15 step of static image data in a moving image in Fig. 9; and 
Fig. 14 is a flowchart showing details of an 
extraction step of input drawing static data in Fig. 9. 

Detailed Description of the Preferred Embodiments 

[0021] 

20 An embodiment of the invention will be described 

below in detail with reference to Figs. 7 to 11. Fig. 7 is 
a diagram showing a drawing processing apparatus according 
to the embodiment of the invention. 

The drawing processing apparatus according to the 
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embodiment of the invention is used in, for example, a 
teleconference system configured by connecting a plurality 
of participant terminals which are used in a conference 
through a communication line. 
5 As shown in Fig. 7, a drawing processing apparatus 

100 shows a configuration for data processing, and has a 
moving image storage section 110 for storing moving image 
data, a moving image reproducing section 111, an image 
control section 112, an image information storage section 

10 113, an image combining section 114 and an image drawing 
section 115. A drawing input device 116 and a screen 
display device 117 are connected to the outside of the 
drawing processing apparatus 100. In the drawing 

processing apparatus 100, a flow of data is shown by arrows 

15 of solid lines and a flow of control signals is shown by 
arrows of dotted lines. 
[0022] 

The drawing processing apparatus 100 has the so- 
called teleconference function and can also handle business 

20 applications such as a spreadsheet, a word processor and a 
presentation. The screen display device 117 of the drawing 
processing apparatus 100 functions as a whiteboard and data 
of applications or data written in the whiteboard is shared 
between a plurality of drawing processing apparatuses 100 

2 5 and a conference can be given with the same contents 
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displayed in these apparatuses. The drawing processing 
apparatus 100 is configured by a general-purpose personal 
computer . 
[0023] 

5 The moving image storage section 110 is configured, 

for example, by a hard disk or a magneto-optical recording 
medium which is a mass storage medium. Plural moving image 
data used in a teleconference etc. is stored therein by an 
image compression method such as MPEG2 , for example, by 
10 encoding. Moving image data which a user selects from 
among the plural moving image data as necessary can be read 
to the moving image reproducing section 111. 
[0024] 

The moving image reproducing section 111 converts the 
15 moving image data selectively read from the moving image 
storage section 110 into reproducible data, for example, by 
decoding. Display in a form capable of perceiving the 
moving image data is not performed in the moving image 
reproducing section 111. 
20 The image control section 112 cuts out the moving 

image data captured to the moving image reproducing section " 
111 every time AT, and outputs the moving image data as 
static image data. 
[0025] 

25 The drawing input device 116 inputs data which a user 
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draws by, for example, a fingertip or an electronic pen. 
This input data may be a point or a line, or drawing data 
in which these point and line are combined, and further 
text data. The image information storage section 113 is 
5 configured by, for example, a storage medium such as 
semiconductor memory. The image information storage 

section 113 stores the drawing data as input drawing data, 
and the image control section 112 extracts the input 
drawing data every time AT. The extracted drawing data is 
10 stored in an area where is different from a storage area of 
the static image data in a moving image in the image 
information storage section 113 as input drawing static 
image data. 
[0026] 

15 The image combining section 114 functions so as to 

combine the static image data in the moving image stored in 
the image information storage section 113 with the input 
drawing static data, and create combined image data as one 
frame forming a moving image in a pseudo manner. 

20 [0027] 

The image drawing section 115 continuously outputs 
the combined image data of a frame unit to the screen 
display device 117 and thereby, a pseudo moving image in 
which the input drawing data is written is displayed on the 
25 screen display device 117. 
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[0028] 

Next, a drawing processing method using the drawing 
processing apparatus according to the embodiment of the 
invention will be explained with reference to an 
5 explanatory diagram showing a flow of image combining 
processing shown in Fig. 8 and a flowchart shown in Fig. 9. 

First, a user selects moving image data required for 
a teleconference from the moving image storage section 110. 
The moving image data selected is converted into 
10 reproducible data, for example, by decoding. In the moving 
image reproducing section 111, a moving image is not 
displayed on the screen display device 117. On the other 
hand, input drawing data which the user inputs from the 
drawing input device 116 is once captured and stored in the 
15 image information storage section 113. 
[0029] 

Subsequently, as shown in Fig . 9, in order to combine 
a moving image with input drawing, moving image data on the 
moving image reproducing section 111 is cut out every 

2 0 predetermined time AT by a cutout control signal outputted 
by the image control section 112 (step SI) . Input drawing 
static data is extracted from input drawing data on the 
image information storage section 113 every time AT by the 
cutout control signal (step S2) . 

25 [0030] 
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The moving image data which is cut out and the input 
drawing static data which is extracted are stored as static 
image data in an moving image and input drawing static data 
in the image information storage section 113, and are read 
5 to overlap each of these data at predetermined timing by 
the image combining section 114, and combined image data is 
created (combined image creation step S3) . Then, the 
combined image data is outputted to the screen display 
device 117 (image output step S4). 
10 [0031] 

Then, it is decided whether the time AT or longer has 
elapsed or not with respect to time At taken to perform 
screen display of the combined image data outputted in the 
image output step S4 after performing cutout processing of 

15 the moving image and the drawn input image through the 
image cutout steps SI and S2 (step S5) . If the time At has 
not elapsed the time AT, the processing is waited. If the 
time At has elapsed the time AT, the processing subsequent 
to the step SI described above is repeated. 

20 [0032] 

In this manner, the combined image data is outputted, 
the processing is waited for the time AT to elapse, and the 
processing subsequent to the step SI is again performed 
repeatedly at high speed. Therefore, a moving image can be 
25 formed in a pseudo manner, and writing of the input drawing 
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data on the moving image can be implemented. 
[0033] 

Thus, as an interval of the time AT is set to be 
slightly longer than or equal to time (At) taken to perform 
5 screen display after performing cutout processing of the 
image, a load can be prevented from being applied to a 
general-purpose PC more than necessary. 
[0034] 

As a result, parallel processing with writing of the 
10 drawing and the processing by the general-purpose PC can be 
performed smoothly. Since the cutout of an image is 
performed in real time at the time of reproduction of the 
moving image data or the input drawing data on the real 
time axis, a deviation from sound also falls within the At 
15 substantially. 
[0035] 

When processing time necessary for the cutout etc. of 
the static image data in a moving image increases, display 
interval time AT of the screen becomes long. However, 
20 since cutout time of the static image data in a moving 
image also shifts with the delay, a situation in which a 
deviation from sound is gradually increasing while 
producing the moving image does not occur. 
[0036] 

25 Next, one example of the image combining processing 
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in the drawing processing method mentioned above will be 
specifically explained with reference to Figs. 10 to 12. 

Input drawing static data is held as a set of drawing 
data represented in a vector format configured by including 
5 color, size, the number of points, a coordinate data set 
and so on. For example, the input drawing static data 
shown in Fig. 10 is configured by drawing data L(l)and 
drawing data L(2). In the drawing data L(l), color is 
blue, size is 3pt, the number of points is m, and a 

10 coordinate data set includes coordinates P(l) to P (m) . In 
the drawing data L(2), color is red, size is lpt, the 
number of points is n, and a coordinate data set includes 
coordinates P(l) to P(n). Points forming the drawing data 
L(l) or L(2) are acquired every event that a mouse has 

15 moved at the time of input drawing by a pointing device 
such as a mouse. 
[0037] 

Then, combination with the static image data in a 
moving image stored in the image information storage 

20 section 113 is performed by writing the drawing data of the 
input drawing static data on the static image data in a 
moving image stored in the image information storage 
section 113. Description will be made below with reference 
to a flowchart of Fig. 11 and one example of combined 

25 images of Figs. 12A to 12C. 
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[0038] 

First, static image data in a moving image is cut out 
by the moving image reproducing section 111 (step S31, see 
Fig. 12A) . Also, input drawing static data is extracted 
5 from input drawing data stored in the image information 
storage section 113 (step S32) . Next, whether or not 
writing is performed in this input drawing static data is 
judged (step S33) . When the writing is not performed, the 
static image data in a moving image is outputted to the 
10 screen display device as it is (step 39) . 
[0039] 

On the other hand, when the writing is performed, "0" 
is first set to a writing count cnt (step S34) . Next, this 
writing count cnt is incremented (addition of "1") (step 
15 S35), then drawing data L (cnt) is overwritten in the 
static image data in a moving image (step S36, see Fig. 12B 
of a combined image for cnt=l) . 
[0040] 

Then, the writing count cnt is compared with the 
2 0 number of drawing data (step S37) . When the writing count 
cnt is more than or equal to the drawing data, that is, 
when all the drawing data (L(l), L(2)) is overwritten in 
the static image data in a moving image (see Fig. 12C of a 
combined image for cnt=2), combining processing is 
25 completed (step S38) . Then, this data of the combined 
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image of Fig. 12C is outputted to the screen display device 

(step S39) . 

[0041] 

As a method for overwriting the drawing data 
5 described above in the static image data in a moving image, 
the drawing data may be mixed with the static image data in 
a moving image at an arbitrary ratio by translucent 
processing such as a blending rather than full overwriting. 
[0042] 

10 Thus, according to the combining method described 

above, since the input drawing static data is held as a set 
of the drawing data, combining time relates to the amount 
of drawing data forming the input drawing static data 
regardless of screen size. 

15 As a result, when the amount of drawing data forming 

the input drawing static data is small, processing time At 
taken to combine an image is also short. Therefore, more 
combined images can be displayed on a screen and a moving 
image can be shown more smoothly. 

20 [0043] 

However, when the amount of drawing data forming the 
input drawing static data increases, a count of overwriting 
to the static image data in a moving image stored in a 
memory increases to result in an increase in processing 
25 time At necessary for combining time. 
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However, it is rare to perform input drawing so much 
that a moving image of a background is invisible in a 
situation of writing on a moving image. Since a tendency 
to suppress the amount of drawing data forming the input 
5 drawing static data to a small amount is had, a demerit 
that processing time At necessary for combining time 
increases with an increase in the amount of drawing data 
presents no problem substantially. 
[0044] 

10 In the embodiment of the image combining processing 

described above, although the input drawing static data is 
a set of drawing data represented in a vector format, the 
format may be a bitmap format. 

When the data of the bitmap format is combined with 

15 the static image data in a moving image, it is necessary to 
perform comparison calculation for respectively determining 
whether or not overwriting is performed with respect to all 
the pixels forming an image. Then, in the pixel determined 
that the overwriting is performed, overwriting processing 

20 is performed by the same color as that of a pixel of 
corresponding coordinates in the static image data in a 
moving image . 

Since the number of pixels increases when screen size 
becomes large, combining processing time At including 
25 determination processing at the time of combining 
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increases . 
[0045] 

According to the embodiment, as described above, 
since moving image data and input drawing data are not on 
5 the same layer, these data can be held individually. 
Therefore, as an image after screen capture can be handled 
as two layers of background video and an input image, the 
range of uses increases. For example, it can also be 
configured that only input drawing data can be deleted from 
10 a combined image captured. 

In the above, although an example of performing 
drawing in moving image data prepared previously has been 
shown, live video captured from a video camera can also be 
used as a source of moving image data. 
15 [0046] 

Since combined image data obtained in the image 
drawing section 115 is a pseudo moving image in which 
static images are continuous as described above, the 
combined image data can be captured by input of a screen 

20 capture signal with a screen capture operation by a user 
(step S6) . Therefore, when a plurality of drawing 
processing apparatuses for participating in a 
teleconference are connected through a communication line 
as participant terminals, information interchange can be 

25 performed interactively inside a setting screen on a PC 
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screen while performing capture of an image or writing of 
drawing data in a moving image used in the conference. 
[0047] 

Fig. 13 is a flowchart showing details of a cutout 
5 step of static image data in a moving image (step SI) of 
Fig. 9. In the cutout of the static image data in a moving 
image, when a user first selects a moving image file (step 
Sll) , memory expansion of the selected moving image file is 
performed (step S12) and moving image data is stopped in a 
10 frame unit (step S13) . 
[0048] 

Next, the processing is waited for a reproducing 
operation of the moving image data by the user. When the 
reproducing operation is performed, the moving image file 

15 is reproduced (step S15) . This reproduced image is not 
outputted to the screen display device 117. However, sound 
of the moving image is outputted. Then, the processing is 
waited for a stop operation by the user, and when the stop 
operation is performed, the processing subsequent to the 

20 step S13 is repeatedly performed again (step S16) . In the 
moving image stop state (step S13) and the reproducing 
state (step S15) of the moving image file, the static image 
data in a moving image is cut out every time AT (step S17) . 
[0049] 

25 Fig. 14 is a flowchart showing details of an 
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extraction step of input drawing static data (step S2) of 
Fig. 9. In the extraction of this input drawing static 
data, input drawing data is first stored in the image 
information storage section 113 (step S21) . When input 
5 (deletion) of drawing by a user is performed (step S22) , 
inputted drawing data is added to the input drawing data 
accumulated (step S23) . The input drawing data stored in 
memory in step S21 is cut out every time AT (step S24) . 
[0050] 

10 Thus, by rapidly repeating that a screen is cut out 

of the moving image at regular intervals without displaying 
a moving image and drawing data is combined thereon to 
display the combined image, a moving image is created in a 
pseudo manner to display this moving image. 

15 [0051] 

Since screen data is captured to an OS once, capture 
of a screen from this screen data can also be implemented 
smoothly. Therefore, in a teleconference system etc., 
since a state of a moving image can be perceived while 
20 writing information, a presentation can be supported 
effectively. 
[0052] 

As described in detail above, the drawing processing 
apparatus 100 according to the embodiment has an image 
25 control section 112 for cutting out an image as static 
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image information in a moving image from the moving image 
every predetermined time and extracting input drawing 
static information from a drawn input image every the 
predetermined time, an image information storage section 
5 113 for storing the static image data in the moving image 
cut out by the image control section 112 and the input 
drawing static information extracted by the image control 
section 112, an image combining section 114 for combining 
the static image information in the moving image and the 

10 input drawing static information stored in the image 
information storage section 113 to create combined image 
information, and an image drawing section 115 for 
continuously outputting the combined image information. 

The drawing processing method according to the 

15 embodiment has an image cutout step SI of cutting out an 
image as static image information in a moving image from 
the moving image, an image extraction step S2 of extracting 
input drawing static information from a drawn input image, 
a combined image creation step S3 of combining the static 

2 0 image information in the moving image obtained by the 
cutout and the input drawing static information obtained by 
the extraction to create combined image information, and an 
image output step S4 of outputting the combined image 
information, wherein the cutout of the static image 

25 information in the moving image and the extraction of the 
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input drawing static information are repeated every 
predetermined time. 

The drawing processing program according to the 
embodiment, causes a computer to perform an image control 
5 function of cutting out an image as static image 
information in a moving image from the moving image every 
predetermined time and extracting input drawing static 
information from a drawn input image every the 
predetermined time, and an image combining function of 

10 combining the static image information in the moving image 
cut out by the image control function and the input drawing 
static information extracted by the image control function 
to create combined image information. 

The teleconference system is configured by using the 

15 drawing processing apparatus 100 as participant terminal 
for participating in a conference. 
[0053] 

Thus, due to repeating actions in which a screen is 
cut out of a moving image every predetermined time and 

20 drawing data is combined thereon to display the combined 
moving image, a function in which a user writes a symbol, 
an arrow, a character, any writing, etc. on a display 
screen by a drawing input device etc. to combine the symbol 
etc. with the moving image and displays in real time can be 

25 balanced with a function in which the combined image is 
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captured to be retained in a storage medium etc. and is 
displayed later. Then, interactive communication of 
drawing data in a teleconference can be implemented. 
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